## witness_surface_before9: 701 observations
## witness_surface_before10: 887 observations
## witness_surface_before11: 1295 observations
## witness_surface_before12: 214 observations
## witness_surface_before13: 1066 observations
## witness_surface_before14: 204 observations
## witness_surface_after9: 292467 observations
## witness_surface_after10: 144961 observations
## witness_surface_after11: 224730 observations
## witness_surface_after12: 121144 observations
## witness_surface_after13: 58732 observations
## witness_surface_after14: 99428 observations
##7.5
## === BRIGHTNESS ANALYSIS INITIALIZATION ===
## Analyzing image brightness patterns across surface and edge types...
## Target analyses: Starshade Surface, Calibration Surface, Witness Surface, Starshade Edge, Calibration Edge
## === SURFACE BRIGHTNESS ANALYSIS ===
## Available surface datasets: Starshade Surface Before, Starshade Surface After, Calibration Surface Before, Calibration Surface After, Witness Surface Before, Witness Surface After
## Processing Starshade Surface Before - Total images: 15111
## Processing Starshade Surface After - Total images: 15120
## Processing Calibration Surface Before - Total images: 6146
## Processing Calibration Surface After - Total images: 7631
## Processing Witness Surface Before - Total images: 1413
## Processing Witness Surface After - Total images: 3010
## Total surface images analyzed: 48431
## Images with transformation applied: 35462
## # A tibble: 6 × 4
## Surface_Type Timing Total_Images Transformed_Images
## <chr> <chr> <int> <int>
## 1 Calibration After 7631 7631
## 2 Calibration Before 6146 6146
## 3 Starshade After 15120 10547
## 4 Starshade Before 15111 8263
## 5 Witness After 3010 2006
## 6 Witness Before 1413 869
## === EDGE BRIGHTNESS ANALYSIS ===
## Available edge datasets: Starshade Edge, Calibration Edge
## Processing Starshade Edge - Total samples: 975
## Extracting trial numbers from column: image_name
## Available brightness columns: Bef_Pre_Upper_Median, Bef_Pre_Lower_Median, Bef_Pre_Entire_Median, Bef_Post_Upper_Median, Bef_Post_Lower_Median, Bef_Post_Entire_Median, Aft_Pre_Upper_Median, Aft_Pre_Lower_Median, Aft_Pre_Entire_Median, Aft_Post_Upper_Median, Aft_Post_Lower_Median, Aft_Post_Entire_Median
## Using brightness columns: Bef_Pre_Upper_Median, Bef_Pre_Lower_Median, Bef_Pre_Entire_Median, Bef_Post_Upper_Median, Bef_Post_Lower_Median, Bef_Post_Entire_Median, Aft_Pre_Upper_Median, Aft_Pre_Lower_Median, Aft_Pre_Entire_Median, Aft_Post_Upper_Median, Aft_Post_Lower_Median, Aft_Post_Entire_Median
## Processing Calibration Edge - Total samples: 994
## Extracting trial numbers from column: image_name
## Available brightness columns: Bef_Pre_Upper_Median, Bef_Pre_Lower_Median, Bef_Pre_Entire_Median, Bef_Post_Upper_Median, Bef_Post_Lower_Median, Bef_Post_Entire_Median, Aft_Pre_Upper_Median, Aft_Pre_Lower_Median, Aft_Pre_Entire_Median, Aft_Post_Upper_Median, Aft_Post_Lower_Median, Aft_Post_Entire_Median
## Using brightness columns: Bef_Pre_Upper_Median, Bef_Pre_Lower_Median, Bef_Pre_Entire_Median, Bef_Post_Upper_Median, Bef_Post_Lower_Median, Bef_Post_Entire_Median, Aft_Pre_Upper_Median, Aft_Pre_Lower_Median, Aft_Pre_Entire_Median, Aft_Post_Upper_Median, Aft_Post_Lower_Median, Aft_Post_Entire_Median
## Total edge measurements analyzed: 23628
## # A tibble: 2 × 3
## Edge_Type Total_Samples Total_Measurements
## <chr> <int> <int>
## 1 Calibration 994 11928
## 2 Starshade 975 11700
## Creating surface brightness visualizations...
## Attempting timing effect analysis...
## Columns created after pivot_wider: Surface_Type, Trial
## Expected columns found:
## Could not find expected columns for timing effect analysis.
## Available columns: Surface_Type, Trial
## Creating edge brightness visualizations...
##
## === SURFACE BRIGHTNESS SUMMARY STATISTICS ===
## # A tibble: 6 × 8
## Surface_Type Timing Count Original_Mean Original_SD PostTransform_Mean
## <chr> <chr> <int> <dbl> <dbl> <dbl>
## 1 Calibration After 7631 3.8 2.7 3.4
## 2 Calibration Before 6146 4.8 2.2 3.5
## 3 Starshade After 15120 2.1 1.6 2
## 4 Starshade Before 15111 1.6 1.6 1.5
## 5 Witness After 3010 2 1.4 2
## 6 Witness Before 1413 1.8 1.5 1.8
## # ℹ 2 more variables: PostTransform_SD <dbl>, Avg_Change_Percent <dbl>
##
## === SURFACE BRIGHTNESS NORMALIZATION EFFECTIVENESS ===
## # A tibble: 6 × 5
## Surface_Type Timing Original_CV PostTransform_CV CV_Improvement_Percent
## <chr> <chr> <dbl> <dbl> <dbl>
## 1 Calibration After 0.708 0.626 11.5
## 2 Calibration Before 0.463 0.632 -36.5
## 3 Starshade After 0.748 0.751 -0.5
## 4 Starshade Before 1.01 1.01 -0.2
## 5 Witness After 0.709 0.709 0
## 6 Witness Before 0.791 0.791 0
##
## === EDGE BRIGHTNESS SUMMARY STATISTICS ===
## # A tibble: 24 × 8
## Edge_Type Contamination Processing Region Count Mean_Brightness SD_Brightness
## <chr> <chr> <chr> <chr> <int> <dbl> <dbl>
## 1 Calibrat… After Contam… After Pro… Entire 994 82.4 78.9
## 2 Calibrat… After Contam… Before Pr… Entire 994 93.6 89.1
## 3 Calibrat… Before Conta… After Pro… Entire 994 154. 60.3
## 4 Calibrat… Before Conta… Before Pr… Entire 994 170. 65
## 5 Calibrat… After Contam… After Pro… Lower 994 8 0
## 6 Calibrat… After Contam… Before Pr… Lower 994 10.5 2
## 7 Calibrat… Before Conta… After Pro… Lower 994 8 0
## 8 Calibrat… Before Conta… Before Pr… Lower 994 10.5 0.8
## 9 Calibrat… After Contam… After Pro… Upper 994 186 0
## 10 Calibrat… After Contam… Before Pr… Upper 994 206. 14.9
## # ℹ 14 more rows
## # ℹ 1 more variable: CV <dbl>
##
## === EDGE BRIGHTNESS CHANGE EFFECTS BY EDGE TYPE ===
## Average brightness change due to contamination:
## # A tibble: 12 × 5
## Edge_Type Processing Region Mean_Contamination_E…¹ SD_Contamination_Eff…²
## <chr> <chr> <chr> <dbl> <dbl>
## 1 Calibration After Proce… Entire -72.1 NA
## 2 Calibration After Proce… Lower 0 NA
## 3 Calibration After Proce… Upper 0 NA
## 4 Calibration Before Proc… Entire -76.1 NA
## 5 Calibration Before Proc… Lower 0 NA
## 6 Calibration Before Proc… Upper 0.9 NA
## 7 Starshade After Proce… Entire -3.6 NA
## 8 Starshade After Proce… Lower 0 NA
## 9 Starshade After Proce… Upper 0 NA
## 10 Starshade Before Proc… Entire -0.7 NA
## 11 Starshade Before Proc… Lower 1.1 NA
## 12 Starshade Before Proc… Upper 5.8 NA
## # ℹ abbreviated names: ¹​Mean_Contamination_Effect, ²​SD_Contamination_Effect
##
## Average brightness change due to processing:
## # A tibble: 12 × 5
## Edge_Type Contamination Region Mean_Processing_Effect SD_Processing_Effect
## <chr> <chr> <chr> <dbl> <dbl>
## 1 Calibration After Contami… Entire -11.3 NA
## 2 Calibration After Contami… Lower -2.5 NA
## 3 Calibration After Contami… Upper -20.2 NA
## 4 Calibration Before Contam… Entire -15.2 NA
## 5 Calibration Before Contam… Lower -2.5 NA
## 6 Calibration Before Contam… Upper -19.2 NA
## 7 Starshade After Contami… Entire -3.4 NA
## 8 Starshade After Contami… Lower -3.8 NA
## 9 Starshade After Contami… Upper -5 NA
## 10 Starshade Before Contam… Entire -0.5 NA
## 11 Starshade Before Contam… Lower -2.6 NA
## 12 Starshade Before Contam… Upper 0.8 NA
##
## === STARSHADE vs CALIBRATION EDGE COMPARISON ===
## # A tibble: 12 × 7
## Contamination Processing Region Calibration Starshade Difference
## <chr> <chr> <chr> <dbl> <dbl> <dbl>
## 1 After Contamination After Processing Entire 82.4 119. 37
## 2 After Contamination After Processing Lower 8 10 2
## 3 After Contamination After Processing Upper 186 200 14
## 4 After Contamination Before Processi… Entire 93.6 123. 29.1
## 5 After Contamination Before Processi… Lower 10.5 13.8 3.3
## 6 After Contamination Before Processi… Upper 206. 205. -1.1
## 7 Before Contamination After Processing Entire 154. 123. -31.6
## 8 Before Contamination After Processing Lower 8 10 2
## 9 Before Contamination After Processing Upper 186 200 14
## 10 Before Contamination Before Processi… Entire 170. 123. -46.3
## 11 Before Contamination Before Processi… Lower 10.5 12.6 2.1
## 12 Before Contamination Before Processi… Upper 205. 199. -6
## # ℹ 1 more variable: Percent_Difference <dbl>
##
## === BRIGHTNESS ANALYSIS COMPLETE ===
## Results stored in 'brightness_analysis_results' list for integration with main analysis.
## Surface types analyzed: Starshade Before, Starshade After, Calibration Before, Calibration After, Witness Before, Witness After
## Edge types analyzed: Starshade, Calibration
##
## Available result components:
## - surface_data: Combined surface brightness data (Starshade, Calibration, Witness × Before/After)
## - edge_data: Combined edge brightness data (Starshade, Calibration)
## - surface_summary: Summary statistics for all surface types and timing
## - edge_summary: Summary statistics for all edge types
## - normalization_effectiveness: CV improvement metrics by surface type and timing
## - contamination_effects: Brightness changes due to contamination by edge type
## - processing_effects: Brightness changes due to processing by edge type
## - edge_comparison: Direct Starshade vs Calibration edge comparison
## - surface_plots: Surface visualization plots (distribution, change_by_transform, before_vs_after, variability, processing_effect, timing_effect, comparison)
## - edge_plots: Edge visualization plots (brightness_by_status, contamination_effect, processing_effect, edge_comparison)
## TableGrob (2 x 1) "arrange": 2 grobs
## z cells name grob
## 1 1 (1-1,1-1) arrange text[GRID.text.5265]
## 2 2 (2-2,1-1) arrange gtable[arrange]
## TableGrob (2 x 1) "arrange": 2 grobs
## z cells name grob
## 1 1 (1-1,1-1) arrange text[GRID.text.6392]
## 2 2 (2-2,1-1) arrange gtable[arrange]
##
## === MISSING DATA LOG ===
## Total missing data entries: 0
## No missing data detected - all datasets loaded successfully!
##
## ========================
## TableGrob (2 x 1) "arrange": 2 grobs
## z cells name grob
## 1 1 (1-1,1-1) arrange text[GRID.text.7504]
## 2 2 (2-2,1-1) arrange gtable[arrange]
## Summary of missing data:
## Trial 9 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`.
## Trial 9 - Edge Cumulative Scatter - ALL: No data lines available - using fallback
## Trial 9 - Edge Binned Scatter - ALL: No data lines available - using fallback
## Trial 10 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`.
## Trial 10 - Edge Cumulative Scatter - ALL: No data lines available - using fallback
## Trial 10 - Edge Binned Scatter - ALL: No data lines available - using fallback
## Trial 11 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`.
## Trial 11 - Edge Cumulative Scatter - ALL: No data lines available - using fallback
## Trial 11 - Edge Binned Scatter - ALL: No data lines available - using fallback
## Trial 12 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`.
## Trial 12 - Edge Cumulative Scatter - ALL: No data lines available - using fallback
## Trial 12 - Edge Binned Scatter - ALL: No data lines available - using fallback
## Trial 13 - Surface Cumulative Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`.
## Trial 13 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`.
## Trial 13 - Edge Cumulative Scatter - ALL: No data lines available - using fallback
## Trial 13 - Edge Binned Scatter - ALL: No data lines available - using fallback
## Trial 14 - Surface Binned Scatter - ERROR: Error occurred: In argument: `valid_scatters = scatter_clean[is.finite(scatter_clean) & scatter_clean > 0]`.
## Trial 14 - Edge Cumulative Scatter - ALL: No data lines available - using fallback
## Trial 14 - Edge Binned Scatter - ALL: No data lines available - using fallback
## Plot list lengths:
## Surface cumulative: 8
## Surface binned: 8
## Edge cumulative: 8
## Edge binned: 8
## Required length: 8
## TableGrob (2 x 1) "arrange": 2 grobs
## z cells name grob
## 1 1 (1-1,1-1) arrange text[GRID.text.8630]
## 2 2 (2-2,1-1) arrange gtable[arrange]
## Grid created successfully!
## [1] "=== INSTRUMENT SIGNATURE GAP PREDICTION ==="
## [1] "Scale: 0.208333 microns/pixel"
## [1] "Area conversion: 0.043403 microns²/pixel²"
## [1] "\n=== SUMMARY COMPARISON ==="
## [1] "Raw Gap Counts (before merging nearby gaps):"
## Scenario Threshold Circle_Gaps Bin_Gaps Threshold_Gaps Scale_Gaps
## 1 Starshade Edge 60 46 3 7 8
## 2 Calibration Edge 40 46 3 7 8
## Raw_Total
## 1 64
## 2 64
## [1] "\nMerged Gap Counts (after combining gaps within 0.05 log units):"
## Scenario Threshold Merged_Total Reduction Percent_Reduction
## 1 Starshade Edge 60 27 37 57.8
## 2 Calibration Edge 40 26 38 59.4
## [1] "\nGaps unique to Starshade Edge: 1"
## [1] "Gaps unique to Calibration Edge: 0"
## [1] "Starshade-specific gaps (first 5):"
## [1] 1.073
## [1] "\n=== ANALYSIS OF OBSERVED GAP ==="
## [1] "Observed gap at log10(area) = 2.3"
## [1] "Area: 199.5 μm²"
## [1] "Diameter: 15.9 μm"
## [1] "Closest Starshade prediction: 0.188 log units away"
## [1] "Closest Calibration prediction: 0.188 log units away"
## [1] "*** Starshade Edge model better predicts your observed gap ***"
## [1] "\n=== ARRAYS FOR RMARKDOWN ==="
## # Starshade Edge Predicted Gaps (Threshold 60)
## starshade_predicted_gaps <- c( 0.038, 0.092, 0.214, 0.339, 0.450, 0.533, 0.612, 0.691, 0.758, 0.884, 0.941, 0.992, 1.073, 1.155, 1.274, 1.328, 1.395, 1.457, 1.515, 1.569, 1.659, 1.736, 1.819, 1.895, 1.989, 2.081, 2.488 )
## # Calibration Edge Predicted Gaps (Threshold 40)
## calibration_predicted_gaps <- c( 0.038, 0.092, 0.214, 0.339, 0.436, 0.538, 0.612, 0.691, 0.758, 0.884, 0.941, 0.992, 1.155, 1.274, 1.328, 1.395, 1.457, 1.515, 1.569, 1.645, 1.747, 1.819, 1.895, 1.989, 2.081, 2.488 )
## [1] "\n=== VALIDATION READY ==="
## [1] "Use validate_gaps_simple() with your edge particle data:"
## [1] "starshade_validation <- validate_gaps_simple(edge_particles_data, 'Area', starshade_predicted_gaps)"
## [1] "calibration_validation <- validate_gaps_simple(calibration_particles_data, 'Area', calibration_predicted_gaps)"
## === CHECKING AVAILABLE DATASETS ===
## Available datasets:
## - surface_binned : TRUE
## - edge_binned : TRUE
## - cw_edge_binned : TRUE
## - model_edge_binned : TRUE
## Creating surface distribution plot...
## Surface distribution plot created successfully
## Creating edge distribution plot...
## Edge distribution plot created successfully
## Creating normalized distribution plot...
## Normalized distribution plot created successfully
## Creating ratio analysis plot...
## Ratio analysis plot created successfully
## === ANALYSIS SUMMARY ===
## Surface Distribution Analysis:
## - PCL: 175 μm²
## - Slope: -1.54
## - Data points: 100
## Edge Distribution Analysis:
## - Total particles: 259
## - Area range: 1 - 5987 μm²
## - Data points: 100
## Ratio Analysis:
## - Mean etched/calibration ratio: 2.56
## - Median ratio: 0.71
## - Maximum ratio: 31.48
## Plots successfully created: 4 out of 4
## ========================